{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# (II) Detection and Picking\n",
    "This notebook demonstrates the use of EQTransformer for performing the earthquake signal detection and seismic phase (P & S) picking on continuous data. Once you have your seismic data - preferentially in mseed format and in individual subfolders for each station- you can perform the detection/picking using the following options:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Option (I) on preprocessed (hdf5) files:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This option is recommended for smaller time periods (a few days to a month). This allows you to test the perfomance and explore the effects of different parameters while the provided hdf5 file makes it easy to access the waveforms.\n",
    "\n",
    "For this option you first need to convert your MiniSeed files for each station into a single hdf5 file and a csv file containting the list of traces in the hdf5 file. If you are using local MiniSeed files you can generate a station_list.json by supplying an absolute path to a directory containing Miniseed files and a station location dictionary using the stationListFromMseed function like the following:\n",
    "```\n",
    "    from EQTransformer.utils.hdf5_maker import stationListFromMseed\n",
    "    mseed_directory = '/Users/username/Downloads/EQTransformer/examples/downloads_mseeds'\n",
    "    station_locations = {\"CA06\": [35.59962, -117.49268, 796.4], \"CA10\": [35.56736, -117.667427, 835.9]}\n",
    "    stationListFromMseed(mseed_directory, station_locations)\n",
    "```\n",
    "You can convert MiniSeed files to a hdf5 file using the following command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n",
      "/Users/mostafamousavi/anaconda3/envs/test1/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "/Users/mostafamousavi/anaconda3/envs/test1/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:527: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "/Users/mostafamousavi/anaconda3/envs/test1/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:528: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "/Users/mostafamousavi/anaconda3/envs/test1/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:529: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "/Users/mostafamousavi/anaconda3/envs/test1/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:530: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "/Users/mostafamousavi/anaconda3/envs/test1/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:535: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "============ Station B921 has 1 chunks of data.\n",
      "============ Station SV08 has 1 chunks of data.\n",
      "  * SV08 (1) .. 20190901 --> 20190902 .. 2 components .. sampling rate: 100.0\n",
      "  * B921 (1) .. 20190901 --> 20190902 .. 3 components .. sampling rate: 100.0\n",
      " Station SV08 had 1 chuncks of data\n",
      "2056 slices were written, 2057.0 were expected.\n",
      "Number of 1-components: 0. Number of 2-components: 1. Number of 3-components: 0.\n",
      "Original samplieng rate: 100.0.\n",
      "============ Station CA06 has 1 chunks of data.\n",
      "  * CA06 (1) .. 20190901 --> 20190902 .. 3 components .. sampling rate: 100.0\n",
      " Station B921 had 1 chuncks of data\n",
      "2056 slices were written, 2057.0 were expected.\n",
      "Number of 1-components: 0. Number of 2-components: 0. Number of 3-components: 1.\n",
      "Original samplieng rate: 100.0.\n",
      " Station CA06 had 1 chuncks of data\n",
      "2056 slices were written, 2057.0 were expected.\n",
      "Number of 1-components: 0. Number of 2-components: 0. Number of 3-components: 1.\n",
      "Original samplieng rate: 100.0.\n"
     ]
    }
   ],
   "source": [
    "from EQTransformer.utils.hdf5_maker import preprocessor\n",
    "preprocessor(mseed_dir='downloads_mseeds', \n",
    "             stations_json='station_list.json', \n",
    "             overlap=0.3,\n",
    "             n_processor=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This will generate one \"station_name.hdf5\" and one \"station_name.csv\" file for each of your stations and put them into a directory named \"mseed_dir+_hdfs\". Then you need to pass the name of the directory containing your hdf5 & CSV files and a model. You can use relatively low threshold values for the detection and picking since EQTransformer is very robust to false positives. Enabling uncertaintiy estimation, outputing probabilities, or plotting all the detected events will slow down the process."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "============================================================================\n",
      "Running EqTransformer  None\n",
      " *** Loading the model ...\n",
      "*** Loading is complete!\n",
      "============================================================================\n",
      " *** /Users/mostafamousavi/Desktop/EQTransformer/examples/detections1 already exists!\n",
      " --> Type (Yes or y) to create a new empty directory! otherwise it will overwrite!   y\n",
      "######### There are files for 3 stations in downloads_mseeds_processed_hdfs directory. #########\n",
      "========= Started working on B921, 1 out of 3 ...\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "  0%|                                                                         | 0/5 [00:00<?, ?it/s]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 40%|██████████████████████████                                       | 2/5 [00:43<01:05, 21.85s/it]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 60%|███████████████████████████████████████                          | 3/5 [00:53<00:36, 18.20s/it]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 80%|████████████████████████████████████████████████████             | 4/5 [01:04<00:15, 15.96s/it]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "100%|█████████████████████████████████████████████████████████████████| 5/5 [01:13<00:00, 14.03s/it]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\n",
      "\n",
      " *** Finished the prediction in: 0 hours and 1 minutes and 15.2 seconds.\n",
      " *** Detected: 1438 events.\n",
      " *** Wrote the results into --> \" /Users/mostafamousavi/Desktop/EQTransformer/examples/detections1/B921_outputs \"\n",
      "========= Started working on CA06, 2 out of 3 ...\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "  0%|                                                                         | 0/5 [00:00<?, ?it/s]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 40%|██████████████████████████                                       | 2/5 [00:13<00:19,  6.58s/it]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 60%|███████████████████████████████████████                          | 3/5 [00:21<00:14,  7.22s/it]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 80%|████████████████████████████████████████████████████             | 4/5 [00:32<00:08,  8.24s/it]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "100%|█████████████████████████████████████████████████████████████████| 5/5 [00:42<00:00,  8.79s/it]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\n",
      "\n",
      " *** Finished the prediction in: 0 hours and 0 minutes and 44.53 seconds.\n",
      " *** Detected: 1357 events.\n",
      " *** Wrote the results into --> \" /Users/mostafamousavi/Desktop/EQTransformer/examples/detections1/CA06_outputs \"\n",
      "========= Started working on SV08, 3 out of 3 ...\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "  0%|                                                                         | 0/5 [00:00<?, ?it/s]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 40%|██████████████████████████                                       | 2/5 [00:15<00:23,  7.97s/it]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 60%|███████████████████████████████████████                          | 3/5 [00:25<00:17,  8.50s/it]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " 80%|████████████████████████████████████████████████████             | 4/5 [00:35<00:08,  8.77s/it]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "100%|█████████████████████████████████████████████████████████████████| 5/5 [00:44<00:00,  8.86s/it]\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\u001b[A\n",
      "\n",
      " *** Finished the prediction in: 0 hours and 0 minutes and 45.85 seconds.\n",
      " *** Detected: 819 events.\n",
      " *** Wrote the results into --> \" /Users/mostafamousavi/Desktop/EQTransformer/examples/detections1/SV08_outputs \"\n"
     ]
    }
   ],
   "source": [
    "from EQTransformer.core.predictor import predictor\n",
    "predictor(input_dir='downloads_mseeds_processed_hdfs',   \n",
    "         input_model='../ModelsAndSampleData/EqT_model.h5',\n",
    "         output_dir='detections1',\n",
    "         estimate_uncertainty=False, \n",
    "         output_probabilities=False,\n",
    "         number_of_sampling=5,\n",
    "         loss_weights=[0.02, 0.40, 0.58],          \n",
    "         detection_threshold=0.3,                \n",
    "         P_threshold=0.1,\n",
    "         S_threshold=0.1, \n",
    "         number_of_plots=10,\n",
    "         plot_mode='time',\n",
    "         batch_size=500,\n",
    "         number_of_cpus=4,\n",
    "         keepPS=False,\n",
    "         spLimit=60) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Option (II) directly on downloaded MiniSeed files:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can perform the detection/picking directly on .mseed files. \n",
    "This save both prerpcessing time and the extra space needed for hdf5 file. However, it can be more memory intensive. So it is recommended when mseed fils are one month long or shorter.\n",
    "This option also does not allow you to estimate the uncertainties, write the prediction probabilities, or use the advantages of having hdf5 files which makes it easy to access the raw event waveforms based on detection results.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "============================================================================\n",
      "Running EqTransformer  None\n",
      " *** Loading the model ...\n",
      "*** Loading is complete!\n",
      "######### There are files for 3 stations in downloads_mseeds directory. #########\n",
      "========= Started working on B921, 1 out of 3 ...\n",
      "20190901T000000Z__20190902T000000Z.mseed\n",
      "\n",
      "\n",
      " *** Finished the prediction in: 0 hours and 1 minutes and 10.13 seconds.\n",
      " *** Detected: 1393 events.\n",
      " *** Wrote the results into --> \" /Users/mostafamousavi/Desktop/EQTransformer/examples/detections2/B921_outputs \"\n",
      "========= Started working on CA06, 2 out of 3 ...\n",
      "20190901T000000Z__20190902T000000Z.mseed\n",
      "\n",
      "\n",
      " *** Finished the prediction in: 0 hours and 0 minutes and 32.47 seconds.\n",
      " *** Detected: 1345 events.\n",
      " *** Wrote the results into --> \" /Users/mostafamousavi/Desktop/EQTransformer/examples/detections2/CA06_outputs \"\n",
      "========= Started working on SV08, 3 out of 3 ...\n",
      "20190901T000000Z__20190902T000000Z.mseed\n",
      "\n",
      "\n",
      " *** Finished the prediction in: 0 hours and 0 minutes and 29.27 seconds.\n",
      " *** Detected: 801 events.\n",
      " *** Wrote the results into --> \" /Users/mostafamousavi/Desktop/EQTransformer/examples/detections2/SV08_outputs \"\n"
     ]
    }
   ],
   "source": [
    "from EQTransformer.core.mseed_predictor import mseed_predictor\n",
    "mseed_predictor(input_dir='downloads_mseeds',   \n",
    "         input_model='../ModelsAndSampleData/EqT_model.h5',\n",
    "         stations_json='station_list.json',\n",
    "         output_dir='detections2',\n",
    "         loss_weights=[0.02, 0.40, 0.58],          \n",
    "         detection_threshold=0.3,                \n",
    "         P_threshold=0.1,\n",
    "         S_threshold=0.1, \n",
    "         number_of_plots=10,\n",
    "         plot_mode='time_frequency',\n",
    "         normalization_mode='std',\n",
    "         batch_size=500,\n",
    "         overlap=0.3,\n",
    "         gpuid=None,\n",
    "         gpu_limit=None) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Prediction outputs for each station will be written in your output directory (i.e. 'detections').\n",
    "\n",
    "'X_report.txt' contains processing info on input parameters used for the detection/picking and final \n",
    "results such as running time, the total number of detected events (these are unique events and duplicated ones have been already removed). \n",
    "\n",
    "'X_prediction_results.csv' contains detection/picking results in the figures folder you can find the plots for the number of events that you specified in the above comment."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
